Convert voluptuous schema to msgspec by abhishekmadan30 · Pull Request #752 · taskcluster/taskgraph

abhishekmadan30 · 2025-08-26T15:16:44Z

No description provided.

src/taskgraph/util/schema.py

docs/concepts/transforms.rst

src/taskgraph/config.py

src/taskgraph/transforms/matrix.py

ahal

Thanks, this is shaping up nicely!

I don't think I have any other major concerns, it's mostly all nits and a minor changes. So I think you can go ahead and start converting Gecko without fear of a major new request (from me at least :p)

docs/concepts/transforms.rst

src/taskgraph/config.py

src/taskgraph/parameters.py

src/taskgraph/transforms/fetch.py

src/taskgraph/transforms/run/__init__.py

src/taskgraph/util/dependencies.py

src/taskgraph/util/schema.py

...utter.project_name}}/taskcluster/{{cookiecutter.project_slug}}_taskgraph/transforms/hello.py

ahal

Nice, thanks for the updates! I don't see any fundamental problems or anything, though I'll reserve the right to request future changes as we test this out in Gecko ;)

I'll avoid approving this for now because I want to hold off landing until we have a working patch for Gecko, but I think you can go ahead and get started on that! Hopefully claude or some clever macros can help with the conversion, because there's going to be a lot of schemas :)

ahal

I didn't get a chance to review the whole thing, but I'll leave the comments I had so far for now and resume tomorrow. Figured you might want to get a head start

docs/tutorials/creating-a-task-graph.rst

src/taskgraph/config.py

ahal

A few more comments, still not all the way through though.

src/taskgraph/parameters.py

ahal · 2025-09-18T17:55:16Z

src/taskgraph/transforms/docker_image.py

+    # List of package tasks this docker image depends on.
+    packages: Optional[List[str]] = None
+    # Information for indexing this build so its artifacts can be discovered.
+    index: Optional[Any] = None


You should import the schema from gecko_taskgraph.transforms.task and reference the index key there when we come across these "cross references".

src/taskgraph/transforms/fetch.py

src/taskgraph/transforms/from_deps.py

src/taskgraph/transforms/notify.py

src/taskgraph/transforms/run/__init__.py

src/taskgraph/transforms/run/run_task.py

src/taskgraph/transforms/task.py

ahal · 2025-09-25T18:05:48Z

Thanks, looks like you have some merge conflicts too. Please rebase and push your latest iteration and hopefully I can do one final review (of just the interdiff this time)

src/taskgraph/transforms/run/__init__.py

src/taskgraph/transforms/docker_image.py

src/taskgraph/transforms/fetch.py

jcristau · 2025-10-24T10:07:53Z

Is there any chance we can do this in a backwards-compatible way, e.g. by leaving existing util/schema.py APIs alone and adding the msgspec schemas as an alternative, so we can convert things piecemeal and avoid massive changes and likely regressions/back and forth? Or is that impossible for some reason?

abhishekmadan30 · 2025-10-24T17:38:56Z

Is there any chance we can do this in a backwards-compatible way, e.g. by leaving existing util/schema.py APIs alone and adding the msgspec schemas as an alternative, so we can convert things piecemeal and avoid massive changes and likely regressions/back and forth? Or is that impossible for some reason?

Yep, I am currently working on it, I think we will go with just a different name for each of the schemas. eg volutpous is still called Schema while msgspec is called Struct and for shared functions, Ill add use_msgspec flag which by default is to false to limit future compatibility issues

ahal · 2025-10-28T20:04:27Z

src/taskgraph/util/schema.py



-def optionally_keyed_by(*arguments):
+def optionally_keyed_by(*arguments, use_msgspec=False):


Oh, I see what you meant now by having separate functions for this.. For some reason I was thinking there'd be an object passed in here we could test, but that obviously isn't the case. I like this approach though.

ahal · 2025-10-28T20:07:33Z

src/taskgraph/util/schema.py

+    omit_defaults=True,
+    rename="kebab",
+    forbid_unknown_fields=True,
+):


Do you like Struct as a name? Do you have any other ideas? I'm not sure if I like it, but I can't really think of anything better.

ahal · 2025-10-28T20:09:35Z

src/taskgraph/transforms/run/__init__.py

-)
+
+# Fetches schema using msgspec
+class FetchesStruct(Struct):


Would you mind splitting all the actual schema conversion work into a separate commit from supporting msgspec in validate_schema? This way it will be easy to prove that the generation works before and after the change.

ofek · 2025-11-04T04:30:36Z

Hello there! I recently became a co-maintainer of msgspec and am looking through open issues referencing the project name. Let me know if I can help 🙂

ahal · 2025-11-04T19:14:05Z

Hi @ofek, thanks for the offer! I think we're good other than we're waiting for a new msgspec release to pick up the Python 3.14 support. As a library ourselves, we don't want to risk breaking our users if they happen to be using 3.14.

We appreciate you stepping in to take over maintenance!

ahal reviewed Aug 27, 2025

View reviewed changes

src/taskgraph/util/schema.py Outdated Show resolved Hide resolved

docs/concepts/transforms.rst Outdated Show resolved Hide resolved

src/taskgraph/config.py Show resolved Hide resolved

abhishekmadan30 marked this pull request as ready for review August 29, 2025 20:35

abhishekmadan30 requested a review from a team as a code owner August 29, 2025 20:35

abhishekmadan30 requested a review from ahal August 29, 2025 20:35

abhishekmadan30 changed the title ~~WIP: Convert voluptuous schema to taskgraph~~ Convert voluptuous schema to taskgraph Sep 2, 2025

abhishekmadan30 changed the title ~~Convert voluptuous schema to taskgraph~~ Convert voluptuous schema to msgspec Sep 2, 2025

ahal reviewed Sep 2, 2025

View reviewed changes

src/taskgraph/transforms/matrix.py Outdated Show resolved Hide resolved

abhishekmadan30 added the BREAKING CHANGE Backwards incompatible request that will require major version bump label Sep 2, 2025

ahal requested changes Sep 3, 2025

View reviewed changes

ahal reviewed Sep 5, 2025

View reviewed changes

ahal self-requested a review September 17, 2025 17:32

abhishekmadan30 force-pushed the schema-conversion branch from 87e0de9 to 750511c Compare September 17, 2025 19:01

ahal requested changes Sep 17, 2025

View reviewed changes

docs/tutorials/creating-a-task-graph.rst Outdated Show resolved Hide resolved

src/taskgraph/config.py Show resolved Hide resolved

src/taskgraph/config.py Show resolved Hide resolved

src/taskgraph/config.py Outdated Show resolved Hide resolved

ahal requested changes Sep 18, 2025

View reviewed changes

ahal reviewed Sep 18, 2025

View reviewed changes

ahal reviewed Sep 22, 2025

View reviewed changes

abhishekmadan30 force-pushed the schema-conversion branch 5 times, most recently from a1e336a to b936ea5 Compare September 29, 2025 22:21

ahal reviewed Oct 1, 2025

View reviewed changes

src/taskgraph/transforms/run/__init__.py Outdated Show resolved Hide resolved

ahal reviewed Oct 1, 2025

View reviewed changes

src/taskgraph/transforms/run/__init__.py Show resolved Hide resolved

ahal reviewed Oct 1, 2025

View reviewed changes

src/taskgraph/transforms/docker_image.py Show resolved Hide resolved

ahal reviewed Oct 1, 2025

View reviewed changes

src/taskgraph/transforms/fetch.py Show resolved Hide resolved

abhishekmadan30 force-pushed the schema-conversion branch 4 times, most recently from 7623612 to 7c1b2ab Compare October 9, 2025 19:50

abhishekmadan30 requested a review from ahal October 9, 2025 20:22

bhearsum mentioned this pull request Oct 24, 2025

feat: add support for parallel kind processing with threads #840

Draft

abhishekmadan30 force-pushed the schema-conversion branch 4 times, most recently from 96adc30 to 4f129da Compare October 27, 2025 14:54

abhishekmadan30 marked this pull request as draft October 27, 2025 15:09

abhishekmadan30 force-pushed the schema-conversion branch 6 times, most recently from 4fcf590 to 75ca904 Compare October 27, 2025 21:11

feat: update schema from voluptuous to msgspec

6fa1be5

abhishekmadan30 force-pushed the schema-conversion branch from 75ca904 to 6fa1be5 Compare October 27, 2025 21:57

ahal reviewed Oct 28, 2025

View reviewed changes

abhishekmadan30 force-pushed the schema-conversion branch from 28fedd3 to 6fa1be5 Compare October 29, 2025 20:36



		def optionally_keyed_by(*arguments):
		def optionally_keyed_by(*arguments, use_msgspec=False):

Conversation

abhishekmadan30 commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahal Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal commented Sep 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jcristau commented Oct 24, 2025

Uh oh!

abhishekmadan30 commented Oct 24, 2025

Uh oh!

ahal Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

ahal Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

ahal Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

ofek commented Nov 4, 2025

Uh oh!

ahal commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

ahal commented Nov 4, 2025 •

edited

Loading